The SDK tiering requirements state that conformance scores should be calculated against applicable tests only, "excluding legacy backward-compatibility tests (unless the SDK claims legacy support)."

Currently, tier-check runs all scenarios regardless of the spec version the SDK targets. For example, an SDK targeting 2025-11-25 is scored against 2025-03-26-only backward-compatibility scenarios (auth/2025-03-26-oauth-metadata-backcompat, auth/2025-03-26-oauth-endpoint-fallback), even though those are legacy tests.

Fix

Add a --spec-version <version> option to tier-check that:

1. Passes --spec-version through to the conformance server and conformance client CLI invocations, so only matching scenarios are executed.
2. Filters the expected scenario list used by reconcileWithExpected, so excluded scenarios aren't counted as failures.
3. Intersects the spec-version filter with the active scenario list (excluding pending scenarios) to avoid counting unimplemented scenarios.

Usage

npx @modelcontextprotocol/conformance tier-check \
  --repo owner/sdk \
  --conformance-server-url http://localhost:8080/mcp \
  --client-cmd "./conformance-client" \
  --spec-version 2025-11-25

Testing

Ran tier-check against my Swift SDK with --spec-version 2025-11-25:

• Without flag: client conformance 18/20 (90%) due to 2 failing 2025-03-26-only backward-compat auth scenarios
• With flag: client conformance 18/18 (100%), server conformance 30/30 (100%)

Breaking Changes

None

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the MCP Documentation
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling (n.a.)
• I have added or updated documentation as needed (n.a.)